Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

--delete-on-faliure recreates cluster for kubeadm failures as well #16890

Merged
merged 1 commit into from
Sep 11, 2023
Merged

--delete-on-faliure recreates cluster for kubeadm failures as well #16890

merged 1 commit into from
Sep 11, 2023

Conversation

ComradeProgrammer
Copy link
Member

feat: allow delete-on-faliure in setupKubeAdm
fix #16682

If we generate an error in setupKubeAdm deliberately, then:

Before:

tjm@tjm:~/workspace/minikube/out$ ./minikube start  --delete-on-failure
😄  minikube v1.30.1 on Ubuntu 22.04 (amd64)
✨  Automatically selected the docker driver
📌  Using Docker driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.27.3 preload ...
    > preloaded-images-k8s-v18-v1...:  393.19 MiB / 393.19 MiB  100.00% 12.13 M
    > index.docker.io/kicbase/bui...:  430.77 MiB / 430.77 MiB  100.00% 8.91 Mi
❗  minikube was unable to download gcr.io/k8s-minikube/kicbase-builds:v0.0.39-1687538068-16731, but successfully downloaded docker.io/kicbase/build:v0.0.39-1687538068-16731 as a fallback image
🔥  Creating docker container (CPUs=2, Memory=2200MB) ...
🐳  Preparing Kubernetes v1.27.3 on Docker 24.0.2 ...

❌  Exiting due to K8S_INSTALL_FAILED: Failed to update cluster: This error is expected

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

When this error happens, minikube exits at once.

After:

tjm@tjm:~/workspace/minikube/out$ ./minikube start --delete-on-failure
😄  minikube v1.30.1 on Ubuntu 22.04 (amd64)
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🏃  Updating the running docker "minikube" container ...
🐳  Preparing Kubernetes v1.27.3 on Docker 24.0.2 .../ E0715 19:19:41.660942   29807 start.go:583] Failed to update cluster: This error is expected

❗  Node  failed to start, deleting and trying again.
🔥  Deleting "minikube" in docker ...
🔥  Deleting container "minikube" ...
🔥  Removing /home/tjm/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=2200MB) ...- ^C

When this error happens, it trys to delete the cluster and retry

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 15, 2023
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ComradeProgrammer
Once this PR has been reviewed and has the lgtm label, please assign medyagh for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 15, 2023
@k8s-ci-robot
Copy link
Contributor

Hi @ComradeProgrammer. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jul 15, 2023
@minikube-bot
Copy link
Collaborator

Can one of the admins verify this patch?

@medyagh
Copy link
Member

medyagh commented Jul 16, 2023

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 16, 2023
@minikube-pr-bot

This comment has been minimized.

Sign in to view
@minikube-pr-bot

This comment has been minimized.

Sign in to view
@medyagh
Copy link
Member

medyagh commented Jul 16, 2023

@ComradeProgrammer can u checkout the testerrorspam ?

seems like it might be adding a spam err in normal start

@minikube-pr-bot

This comment has been minimized.

Sign in to view
@ComradeProgrammer
Copy link
Member Author

ComradeProgrammer commented Jul 17, 2023 •
edited
Loading

@ComradeProgrammer can u checkout the testerrorspam ?

seems like it might be adding a spam err in normal start

It's problem with minikube's version. My makefile is too old

I have rebased the PR so that the version number in the newest makefile will be used

I think it will be ok soon

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 16890) |
+----------------+----------+---------------------+
| minikube start | 52.3s    | 54.2s               |
| enable ingress | 28.1s    | 28.0s               |
+----------------+----------+---------------------+

Times for minikube start: 53.9s 51.8s 51.5s 52.1s 52.2s
Times for minikube (PR 16890) start: 52.7s 55.7s 52.8s 54.6s 55.1s

Times for minikube ingress: 27.8s 28.2s 27.8s 28.3s 28.3s
Times for minikube (PR 16890) ingress: 27.8s 28.8s 27.3s 27.8s 28.3s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 16890) |
+----------------+----------+---------------------+
| minikube start | 23.4s    | 23.8s               |
| enable ingress | 48.5s    | 48.8s               |
+----------------+----------+---------------------+

Times for minikube (PR 16890) start: 23.3s 25.1s 22.9s 25.0s 22.7s
Times for minikube start: 23.2s 23.7s 22.8s 22.6s 25.0s

Times for minikube ingress: 48.4s 48.4s 48.9s 47.9s 48.9s
Times for minikube (PR 16890) ingress: 48.4s 49.4s 49.4s 48.4s 48.4s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 16890) |
+----------------+----------+---------------------+
| minikube start | 21.3s    | 22.5s               |
| enable ingress | 40.9s    | 29.0s               |
+----------------+----------+---------------------+

Times for minikube start: 22.3s 20.6s 21.3s 20.8s 21.4s
Times for minikube (PR 16890) start: 21.1s 24.0s 24.1s 20.2s 22.9s

Times for minikube (PR 16890) ingress: 18.9s 32.4s 30.9s 31.4s 31.4s
Times for minikube ingress: 31.4s 31.4s 30.9s 31.4s 79.4s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
KVM_Linux_containerd TestErrorSpam/setup (gopogh) 0.58 (chart)
KVM_Linux TestErrorSpam/setup (gopogh) 0.58 (chart)
KVM_Linux_crio TestErrorSpam/setup (gopogh) 0.59 (chart)
Hyperkit_macOS TestErrorSpam/setup (gopogh) 0.60 (chart)
Docker_Linux_crio TestFunctional/parallel/ImageCommands/ImageTagAndLoadDaemon (gopogh) 2.94 (chart)
Hyperkit_macOS TestNetworkPlugins/group/custom-flannel/Start (gopogh) 4.19 (chart)
Hyperkit_macOS TestMinikubeProfile (gopogh) 7.78 (chart)
Docker_Linux_containerd_arm64 TestDockerEnvContainerd (gopogh) 10.00 (chart)
Docker_Linux_crio TestPause/serial/SecondStartNoReconfiguration (gopogh) 14.12 (chart)
Docker_macOS TestErrorSpam/setup (gopogh) 20.86 (chart)
Docker_Linux_containerd_arm64 TestErrorSpam/setup (gopogh) 21.05 (chart)
Docker_Linux_crio_arm64 TestErrorSpam/setup (gopogh) 21.05 (chart)
Docker_Linux_docker_arm64 TestErrorSpam/setup (gopogh) 21.05 (chart)
Docker_Linux TestErrorSpam/setup (gopogh) 21.05 (chart)
Docker_Linux_containerd TestErrorSpam/setup (gopogh) 21.18 (chart)
Docker_Linux_crio TestErrorSpam/setup (gopogh) 21.18 (chart)

To see the flake rates of all tests by environment, click here.

@medyagh medyagh merged commit fbd7b71 into kubernetes:master Sep 11, 2023
@medyagh medyagh changed the title feat: allow delete-on-faliure in setupKubeAdm --delete-on-faliure recreates cluster for kubeadm failures as well Sep 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@medyagh medyagh Awaiting requested review from medyagh

@spowelljr spowelljr Awaiting requested review from spowelljr

Assignees
No one assigned
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

If deleteOnFailure flag is specified minikube should delete and recreate
5 participants
@ComradeProgrammer @k8s-ci-robot @minikube-bot @medyagh @minikube-pr-bot